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Comparative Reading Comprehension Skills of grade 4 arid_7 Students 
in a Chicago Mas tery Learning Reading Program* 1985-86 
(a supplement to a longitudinal study of the grade 5 cohort) 



Background 

A Chicago Mastery Learning Reading (CMLR) program was introduced at Kettleby 
Pi Si ±n Sept* i983i Monitoring of one of the orignal student cohorts (grade 3 ) 
was begun in Aprit^t984i The intention then and in the next two years (when 
the cohort was in grades 4, and then 5) was to see whether there were 
significant differences in reading comprehension performance between the CMLR 
cohort and a matched (on IQ and community characteristics) group of students in 
several other nearby schools. 

The attainments (as measured by a sequence of Cloze tests and CTBS) of these 
students were reported annually. The CMLR grade 3 group scored initially 
higher than, then lower than, then equal to^ the comparison group* The report 
Performance of Grade 5 students In a_ CMLR program bh the CTBS Test of Reading 
Comprehension, October 1985 (Feb. 1986) reviews findings over the years. The 
report offers possible explanations of the observations as well as analyses of 
achievement differences by "skill objectives," as defined by the Canadian Tests 
of Basic Skills (CTBS). 

That report also recommended that because of the inconclusiveness of the study 
of one CMLR cohort, that the inquiry be extended to two other grades that had 
also been taught with the Chicago materials and methods. Thus the autumn 1985 
scores attained on the Reading Comprehension Battery of the CTBS by the grade 4 
and grade 7 students at Kettleby and the comparison schools were examined. 

Findings (a) Grade 4 

The grade 4 cohort at Kettleby P.S.is small; 15 students took the CTBS Reading 
Comprehension battery in October 1985. 

The comparison group was drawn from the four YRBE elementary schools that share 
attendance boundaries with Kettleby. 

Similar socioeconomic conditions are to be found across the catchment areas of 
these five schools although kettleby is among the somewhat more "rural". All 
students , including those in the comparison schools were in "mainstream" 
programs (i.e., not in classes for the gifted or French immersion; such 
students had been included in the original study). Iti previous reading 
testing, two of the schools had averages generally equal to or greater than 
Kettleby* s while two had somewhat lower mean averages. 

While the original study groups (grade 3 students iii 1983-84) had been matched 
otl _ I Q ? cores attained in the initial year of the study, no such matching was 
attempted in the current inquiry. 

This is what the 1985 test records show. 
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Table 1: — Grade 4 CTBS Reading Comprehension scores, Autumn ^g8^ 

No. Average Grade 



"SchoolC *)^ Students Score* 



Equival ent 



Kettleby (1 class) 15 20.6 4.0 

Comparison group 115 21.3 4.0 

(4 schools, 5 classes) 

All YRBE grade 4 students 3*021 23.0 4.2 

* On an individual basis, a score of 21 corresponds to the typical average 
score for an October administration; foran autumn administration, a score of 
22 equals a 4.2 G.E. or 50 percentile. For "school averages" the scaling is 
somewhat different (see foot of page). 

this table includes the results for students whose test papers were "flagged" 
(marked separately and scores not included in the YRBE school or summary 
reports)^ "Flagged" test papers include, for example* Special Education or ESL 
students or students who may have been ill ^ust before or during the testing 
period. There were four such "flagged" students in this study, all at one 
school* Their mean scores averaged 18.5 but only <<ne student had a very low 
score* There were a total of 96 students "flagged" for the YRBE grade 4 
reading comprehension test. Their average score was 16.3. A recalculation was 
made for Table 1 when it became clear that there is considerable difference in 
this "flagging" practice among the schools in this study. Kettleby and two 
other schools had scores that suggest that some of their students are de facto 
Special Education cases but none were "flagged". 

The score difference between the Kettleby and comparison group was found to be 
not statistically significant at the 0.05 level. Moreover* it would be diffi- 
cult to maintain that a difference of 0.7 of an item on a 49-item test was of 
practical significance. The Kettleby mean average score is about where it has 
historically been with respect to these five schools, the "middle of the pack"* 

Table 2: Rank order o f s c hools, grade -4^test results 



School 

SchoQl Score Percentile 

A 25.0 69 

B 24.8 69 

National and Regional mean 23.0 53 

Kettleby 20.6 37 

? 19.1 24 

D 17.4 14 

The "School Percentile" figures show the relative status of a school's mean 
grade average compared with all the averages of the schools in the norm 
population. Thus a score at the 69 percentile indicates that the grade 4 
students at schools A and B had, on average, achieved as high or higher than 
69% of schools in the national norm sample, these data help us to appreciate 
the variance within our study group. 
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Additional observations carl be made with either. the Regional mean average score 
(23.0) or the range of mean scores in this population (7.65 as the focus for 
study. 

This distribution of school mean scores suggests not only great achievement 
differences within the group of schools, but significant variations from the 
Regional norm at each extreme (25.0, 17.4). Given that the schools are in 
close proximity (geographically, socioeconomically) , differences on this scale 
may warrant further inquiry* 

Comparison (of school averages) with the national hormfbr ah autumn adminis- 
tration (23 raw score points - 4.2 grade equivalent - 53 percentile) adds the 
observation that Kettleby's grade 4 students and those in two of its neighbours 
likely scored on average below the national norm, even if allowance is made for 
the inclusion of Special Education youngsters (hot included in the CTBS norm 
population) • 



Item analysis was not called for in the recommendation [ because such_irifbrma-_ 
t ^ on »_^? c * a ? 8ailc * 8C hoolj> 1**3 already been reported for the grade 4 arid 7 
students. When the 1985 grade 5 groups were studied, analyses of by-school 
P? r ? (),rmaT1<:e sb^fd s ome differences in the response patterns (related to skills 
objectives) between the CMLR cohort and comparison group. 

In suDiary, there is no evidence that the CMLR program and materials make a 
meaningful difference in the reading comprehension skills of these Kettleby 
students by the beginning of their grade 4 year. 

(b) Grade 7 

The grade 7 cohort at Kettleby consists of one class of 28 students that sat 
for the CTBS reading test in the autumn of i»85s 

The comparison group was again drawn from schools that share attendance 
boundaries. However, because of a Senior Public School organization south of 
Kettleby^ the total catchment area in that sector is somewhat greater than for 
the Junior Public School in the grade 4 studyi To offset the effect of a 
greater number of students from the south-westerly Senior Pi Si, grade 7 stu- 
dents from two additional schools (one a Senior Pi Si) to the southeast and *ast 
of Kettleby were added. In all, there were five schools, 16 classes, and 264 
students in the comparison group. 

As with the grade 4 study, the test data to be presented have been recalculated 
to include "flagged" students. In bur study group of six schools there were 
only three "flagged" students - % again all in one school. For the whole system 
there were some 57 "flagged" grade 7 students. (The impact on scores of inclu- 
sion of "flagged" students is negligible atihe system arid comparison group 
level, about 0.2 of a raw score point on a 57-item test. The impact on the one 
school in the study was 0.6, riot enough to change its relative position within 
the six schools). 
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Table 3: Grade 7 CTBS Reading Comprehension scores, Autumtt 1985 



"School(s)" 

kettleby (1 class) 

Comparison group 
(5 schools! 10 classes) 



All YRBE grade 7 students 



No* 
Students 

28 

267 



3,111 



Average 
Score 

33.5 

31.6 

31.1 



Grade 
Equiv a lent 

7.6 

7;4 

7.2 



The difference between the mean scores of the kettleby and comparison group was 
tested for signifance and there was no statistical significane at the 0.05 
level. Again, as in the grade 4 study, the standard deviations were large and 
the numbers in the study relatively small. Hence, a large score difference was 
needed before we could assert that such a difference could only be expected to 
happen 5 times in 100 by chance along. Another way to put the differences into 
perspective is to observe that the standard error of measurement for ah 
individual pupil on this test is about 2 raw score points. Nonetheless, the 
Kettleby average score puts it into top quartile, as we see in Table 2. 



How close to the 
group? 



'middle of the pack" were the Kettleby grade 7 students as a 



table 4: Rank order schools , grade 7 test re s ult s 



School* 
C 

Kettleby 
E 
F 

Regional Mean 
National Mean 

G 

D 



Avg^ -Score 

34.8 
33.5 
31.9 
31.2 
31.1 
31.Q 
30.5 
30.0 



School Percenti le 

89 
79 
59 

52 
52 
52 
39 
34 



* Schools A and B in the grade 4 study do not have grade 7; many pupils from A 
and B subsequently attend School E. 

As Table 4 suggests, there were very creditable performances at the two highest 
scoring schools, Kettleby included. There is rib manageable way of teasing out 
the contribution of CMLR to the Kettleby students 9 perfbrraaricei However, we 
can note that_ the performance of students at school C was higher than at 
Kettleby at the grade 4 level in 1982, i.e., before CMfcR's introductions No 
attempt has been made to determine to what extent the current grade 7 classes 
have changed (e.g., due to transfer-outs or admissions) or how programs have 
changed at School C (they assuredly have) since there is no relative ranking 
change to account for. 



6 



-5- 



Again we c n note the considerable variance within a relatively small and 
homogeneous community. An average difference of almost 5 raw score pditits on a 
57-item test merits attentions 



Kettleby 1 s grade 7 reading scores, if not statistically significant from the 
mean of the region or national or local comparison group, are in the superior 
range on this occasion. So* we must remind ourselves, were the kettleby grade 
3 students' reading scores in 1984, In the subsequent years those scores 
dipped toward the low end of the distribution (1984-1985) before returning to 
the "middle of the pack" (1985-19865. 

Overview and c onclusion 



What can we make of this supplementary excursion into two other GMfcR cohorts at 
Kettleby? Given the up, down, regress-to- the mean pattern for the initial 
study group (grade 3 in 1983-1984), we should not perhaps be surprised to find 
the present grade 4 in the middle of the distribution (and a bit below the 
regional and national norms) and, at the same time fitri the Kettleby grade 7 
8t:ucleilt8 we ti above the national, regional and comparison group achievement 
levels* 

the apparent inconsistency or randomness suggests that the impact (if any) of 
GMLR on reading comprehension has to be looked for by other means* These would 
not necessarily exclude standardized testing. Certainly something more than we 
have been able to do with current test data seems required if there is any wish 
to lower the level of uncertainty about the worth of CMLR. Whether more 
amibitibus inquiry^ is warranted may depend on how kettleby staff now feel about 
their experience with CMLR* 
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